A test of significance in functional quadratic regression |^ 



Lajos Horvath and Ron Reeder 

Department of Mathematics, University of Utah 
Abstract 

We consider a quadratic functional regression model in which a scalar response depends on a 
functional predictor; the common functional linear model is a special case. We wish to test 
the significance of the nonlinear term in the model. We develop a testing method which is 
based on projecting the observations onto a suitably chosen finite dimensional space using 
functional principal component analysis. The asymptotic behavior of our testing procedure 
is established. A simulation study shows that the testing procedure has good size and power 
with finite sample sizes. We then apply our test to a data set provided by Tecator, which 
consists of near-infrared absorbance spectra and fat content of meat. 

Keywords: Absorption spectra; Asymptotics; Functional data analysis; Polynomial regression; 
Prediction; Principal component analysis. 



1 Introduction and results 

In a predictive model, it may be more natural and appropriate for certain quantities to be 



represented as trajectories rather than a single number (Kirkpatrick and Heckman, 1989). 
For example, a young animal's size may be considered as a function of time, giving a growth 
trajectory. A model to predict a certain response from growth trajectories is useful to animal 
breeders because they may be able to produce more valuable animals by changing their growth 



patterns (Fitzhugh, 1976). Miiller and Zhang (2005 ) used egg-laying trajectories from Mediter 



ranean fruit flies to predict a female fly's remaining lifetime. Frank and Friedman (1993) and 



Wold (1993) provide an early discussion on the applications of principal components to ana- 



lyze curves in chemistry. Yao and Miiller (2010) and Borggaard and Thodberg (1992) used 



absorbance trajectories to predict the fat content of meat samples. The absorbance at any 
particular wavelength is a measurement related to the proportion of light that passes through 
a meat sample. A representative sample of 15 of the 240 absorbance trajectories are pictured 
in Figure [1} 



In functional regression, special attention has been given to functional linear models ( Car- 



dot et al.[ 


2003; 


Shen and Faraway 


2004 


( 


]!ai and Hall 


2006 


However, it 


is pointed out in 


Yao and Miiller 


(2010 


) that this moc 



Hall and Horowitz 2007). 



the regression relationship that may not be appropriate in some scenarios. |Yao and Miiller 
(2010) generalized this to a functional polynomial model, which has greater flexibility. In 



functional polynomial regression, as in standard polynomial regression, one must balance the 
costs and benefits of using more parameters in the model. In this paper, we will develop a 
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Figure 1: Absorbance trajectories from 15 samples of finely chopped pure meat. 

test to determine if a quadratic term is justified in the model or if a functional linear model 
adequately describes the regression relationship. 

The functional quadratic model in which a scalar response, Yn, is paired with a functional 
predictor, X„(t), is defined as 



1 /.I 



h{s,t)X^{s)X^{t) dtds + En, 



(1.1) 



where X^{t) = X„(t) — (X„(t)) is the centered predictor process. If h{s,t) = 0, then 



= E{Yn) and (1.1) reduces to the functional linear model 



1 

K, =/i+ / k{t)X^{t) dt + En. 

'o 



'1.21 



Cardot and Sarda (2011) and Mas and Pumo (2011) point out in their survey papers that 



since we can choose a function in (1.2), the functional linear model can be used in a large 



variety of applications. The functional linear model provides a very simple relation between 



Xn{t) and Yn, so it is important to check if the more involved quadratic model (1.1) provides 



a real improvement. In other words, one should test whether the quadratic term is really 

is, 

(1.3) 

HA:h{s,t)^0. 



needed. To test the significance of the quadratic term in (1.1), we test the null hypothesis. 

Ho : his,t) = 0, 

against the alternative 
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To reduce the dimensionality and avoid overfitting in our functional regression model, we will 
project the predictor process onto a suitably chosen finite dimensional space. The space is 
spanned by the eigenfunctions of C{t, s) = E{Xnit) — fixit)){Xn{s) — fix{s)), the covariance 
function of the predictor process, where fixit) = EXnit). We will denote the eigenfunctions 
and associated eigenvalues by {(t>j(t), Aj), 1 <i < oo}. We can and will assume that Aj is 
the i^^ largest eigenvalue and that the eigenfunctions are orthonormal. It is clear that we can 
assume that h is symmetric, and we also impose the condition that the kernels are in L^: 



h{s, t) = h{t, s) and 



1 rl 



JO 



h'^{s, t)dtds < oo. 



f P{t)dt < oo. 
Jo 



;i-4) 

;i.5) 



Thus we have the expansions 

oo oo 



1=1 j=i 



oo oo 



(1.6) 



i=l j=i+l 



i=l 



and 



(1.7) 



i=l 



By projecting onto the space spanned by {vi, . . . ,Vp} and using (1.6) and (1.7), we can 
write the model (1.1) as 

p p p 

= /Z + 5^ h{X^^, V.) + = J})a^,J V.) {XI V,) + 4, (1.8) 

i=l j=i 



1=1 



where 



oo oo 



p oo 



4 = 5.+ ^ h{X:,v,)+J2 Yl(^-l{^=J})a^AK,v^){K,VJ)+Yl E ^a.AK,v^){K,v,)■ 

i=p+l i=p+l j=i i=l j=p+l 

We note that ( |1.8 ) is written as a standard linear model, but the error term, £*, and the 
design points, {{X^,Vi), I < i < p}, are dependent. 

Unfortunately, we cannot use (1.8) directly for statistical inference since Vi(t) and ^x(t) 
are unknown. We estimate ^x{t) and C{t, s) with the corresponding empiricals 

N 



n=l 



and 



1 ^ 



n=l 
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The eigenvalues and the corresponding eigenf unctions of C(t,s) are denoted by Ai > A2 > • • • 
and vi,V2, ■ ■ ■ ■ Eigenfunctions corresponding to unique eigenvalues are uniquely determined 
up to signs. For this reason, we cannot expect more than to have CiViit) be close to Viit), 
where the q's are random signs. We replace equation (1.8) with 
p p p 

Yn = ^+Y^h{Xn-X,Cit^ + Y^Y^{2-l{i = j})a,,j{Xn-X,cA){Xn-X,c,v,)+e*^ (1.9) 

1=1 i=l j=i 

where 

p p 



i) +^bi{X - IJ,X,CiVi) 



i=l 
P P 



i=l 



5^5^(2 - l{z = j})a,, ((X„ - X,cA){X^ - X,c,%) - {X:,v,){X^^,v,)) . 

i=l j=i 



We can write (1.9) in the concise form 



where 



Y — (Yi, Y2, . . . , Yat) , 
B = (0161,0262, • • ■,Cpbpy 



and 




;i.io) 



A = vech [{ciCjaij (2 — l{i = j}) , 1 < i < j < p}^) 

fc — yti ,b2 , ■ ■ ■ ,tj^ J , 



with 



/Df Ff 1\ 
1 



V 



D„ = vech ({({)„ X„ - X){vj, Xn-X), l<t<j< pf) , 

F„= ((X„-X,t)i), (X„-X,{)2), (X„-X,{)p))^. 

The half-vectorization, vech(-), stacks the columns of the lower triangular portion of the 
matrix under each other. Although we write our model in the form of a general linear model, 
it is important to note that it is not a classical linear model. First, £** is correlated with Z 
because £** contains additional error terms which come from projecting onto a p-dimensional 
space. Another important difference between (1.10) and a classical linear model is that the 
parameters to be estimated, A and B, are random; they depend on the random signs, q. We 
estimate A, B, and fi using the least squares estimator: 




-1 



Z' Z ) Z^Y. 



(1.11) 
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To represent elements of A and B, we will use the notation that A = vech({ajj(2 — l{i — 

j}),l<^< j<pF) andB= (61, S^, 

We expect, under Hq, that A will be close to zero since A is zero. If Hq is not correct, we 
expect the magnitude of A to be relatively large. This suggests that a testing procedure could 
be based on A. Due to the random signs coming from the estimation of the eigenfunctions, A 
will not be asymptotically normal. However, if the random signs are "taken out," asymptotic 
normality can be established. Hence our test statistic will be a quadratic form of A with some 
random weight matrices. Let 

n=l 
1 ^ 
n=l 

and 

1 ^ 

n=l 

where 

p p p 

i=l 1=1 j=i 

are the residuals under Hq. We reject the null hypothesis if 

= ^A^(G - MM^)A 

is large. The main result of this paper is the asymptotic distribution of Un under the null 
hypothesis. First, we discuss the assumptions needed to establish asymptotics for Uisf: 

Assumption 1.1. {Xn{t),n > 1} is a sequence of independent, identically distributed Gaus- 
sian processes. 

Assumption 1.2. 

E (^^ Xl{t) dt^ < 00. 

Assumption 1.3. {£„} is a sequence of independent, identically distributed random variables 
satisfying Esn = and Es^ < 00, 

and 

Assumption 1.4. the sequences {e„} and {X„(t)} are independent. 

The last condition is standard in functional data analysis. It implies that the eigenfunctions 
vi,V2, ■ ■ ■ ,Vp are unique up to a sign. 
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Assumption 1.5. 

Ai > A2 > ■ ■ ■ > Ap+i. 



Theorem 1.1. If Hq, (1.5) and Assumptions 1.1-1.5 are satisfied, then 



where r = p{p + l)/2 is the dimension of the vector A. 
The proof of Theorem lA is given in Section |4} 



Remark 1.1. By the Karhunen-Loeve expansion, every centered, square integrable process, 
X^(t), can be written as 

00 

x^^{t) = Y,U,mit), 
1=1 

where (pi are orthonormal functions. Assumption \l.l\ can be replaced with the requirement that 
C,n,i, in,2, ■ ■ •; in,p CL^c independent with ES^^t = and EC,n,e = for all 1 < i < p. 

Our last result provides a simple condition for the consistency of the test based on Un. 
Let A = vech({ajj(2 — l{i = j}), I < i < j < p}"^), i.e. the first r = p{p + l)/2 coefficients 



in the expansion of h in (1.6) 



Theorem 1.2. // (I.4), {1-5), Assumptions 1.1-1.5 are satisfied and A 7^ 0, then we have 
that 

TT P. 

Un — > 00. 

The condition A 7^ means that h is not the function in the space spanned by the 
functions Vi{t)vj{s), I < i, j < p. 

2 A simulation study 

In this section, we investigate the empirical size and power of the testing procedure for finite 
sample sizes. Seeking to obtain a test of size a = .01, .05, or .10, a rejection region was chosen 
according to the limiting distribution of the test statistic. Since the limiting distribution is 
X^(t'), the rejection region is (A, 00), where P (x^(^) > A) = a. Simulated data was then used 
to compute the outcome of the test statistic. Iterating this procedure 5,000 times, we kept 
track of the proportion of times that the outcome fell in the predetermined rejection region. 
When simulations are done under Hq, this gives us the empirical size of the test, which we 
expect to be close to the nominal size, a, for large sample sizes. When simulations are done 
under the alternative, H^, the proportion gives us the empirical power of the test. 

In our first simulation study, the e„'s were generated according to the distribution of 
independent standard normals. We generated the X„(t)'s according to the distribution of 
independent standard Brownian motions. Then, using k{t) = 1 and h{s,t) = c, we obtained 



Yn according to (1.1 ). Thus the power of the test is a function of the parameter c. In particular, 
when c = 0, the null hypothesis is true. The resulting empirical size and power are given in 
Table E 
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The distribution of our test statistic has been shown to converge to a x^(r). Thus we 
expect the empirical and nominal size to be close for samples of size = 200 and even closer 
when N = 500, as observed in Table [T] Since our testing procedure depends on the choice of 
how many principal components to keep, results are given in Table [T] for p = 1, 2, and 3. One 
possible method of selecting p is to follow the advice of Ramsay and Silverman (2005) and 
choose p so that approximately 85% of the variance within a sample is described by the first 
p principal components. 

Although Theorem is proven under the assumption that X„(t) is a Gaussian process, 
the result of Theorem 11.11 holds under relaxed conditions as discussed in Remark 11.11 We will 
now investigate the empirical size and power of our test when is not a Gaussian pro- 

cess. We generate the e„'s according to a uniform distribution on (—0.5, 0.5). The predictors, 
are generated according to X„(t) = (Ti,„ + T2,„t + T3,„(2t2 - 1) + T^^ni-it^ - 3t)) /4, 
where {Tj^„, 1 < i < 4, 1 < ra} are iid random variables having a t-distribution with 5 degrees 
of freedom. The polynomials in the definition of Xn{t) are the orthogonal Chebyshev polyno- 
mials. The resulting empirical size and power are given in Table |2} We see from Table |2] that 
our testing procedure is robust against non-Gaussian observations. Comparing Tables [T] and 
[2| we see that the value of the test statistics tends to be larger if the X„'s are not normally 
distributed for small A^. The overrejection fades as gets larger so in case of non-Gaussian 
X„'s, larger sample sizes are needed. This also explains the somewhat better power of the 
procedure in the case of non-Gaussian errors. 



3 Application to spectral data 

In this section we apply our test to the data set collected by Tecator and available at 



http://lib.stat.cmu.edu/datasets/tecator. Tecator used 240 samples of finely chopped pure 



meat with different fat contents. For each sample of meat, a 100 channel spectrum of ab- 
sorbances was recorded using a Tecator Infratec food and feed analyzer. These absorbances 
can be thought of as a discrete approximation to the continuous record, Also, for each 

sample of meat, the fat content, Yn was measured by analytic chemistry. 

The absorbance curve measured from the n^^ meat sample is given by = log^^Q [Iq/I), 

where t is the wavelength of the light, Iq is the intensity of the light before passing through the 
meat sample, and / is the intensity of the light after it passes through the meat sample. The 
Tecator Infratec food and feed analyzer measured absorbance at 100 different wavelengths 
between 850 and 1050 nanometers. This gives the values of Xn{t) on a discrete grid from 
which we can use cubic splines to interpolate the values anywhere within the interval. A 
representative sample of 15 of the 240 absorbance trajectories are pictured in Figure [1} 



Yao and Miiller (2010) proposed using a functional quadratic model to predict the fat 



content, F„, of a meat sample based on its absorbance spectrum, Xn{t). We are interested in 



determining whether the quadratic term in (1.1) is needed by testing its significance for this 



data set. From the data, we calculate U240. The p-value is then P{x^{r) > t/240). The test 
statistic and hence the p-value are influenced by the number of principal components that we 



choose to keep. If we select p according to the advice of Ramsay and Silverman (2005), we will 
keep only p = 1 principal component because this explains more than 85% of the variation 
between absorbance curves in the sample. Table [3] gives p- values obtained using p = 1, 2, 
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and 3 principal components, which strongly supports that the quadratic regression provides 
a better model for the Tecator data. 



Table 1: Empirical power of test (in %) based on 5,000 simulations using iid Brownian motions 
for Xn{t) and iid standard normals for £„. 



c 


a = .01 


A^ = 200 


A^ = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


1.02 


1.37 


1.95 


1.10 


1.30 


1.15 


0.2 


10.81 


6.87 


6.52 


30.35 


20.35 


12.85 


0.4 


49.51 


37.24 


29.76 


91.90 


84.25 


74.35 


0.6 


86.68 


77.74 


70.19 


100.00 


99.70 


98.75 


0.8 


98.50 


96.05 


92.98 


100.00 


100.00 


100.00 


1.0 


99.94 


99.57 


99.05 


100.00 


100.00 


100.00 



c 


a= .05 


= 200 


A^ = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


5.15 


6.00 


7.44 


5.60 


5.75 


6.05 


0.2 


25.90 


19.17 


18.02 


53.05 


40.00 


31.35 


0.4 


72.10 


60.31 


50.38 


97.90 


93.70 


88.55 


0.6 


95.21 


90.43 


85.77 


100.00 


99.90 


99.60 


0.8 


99.60 


98.90 


97.60 


100.00 


100.00 


100.00 


1.0 


99.99 


99.87 


99.84 


100.00 


100.00 


100.00 



c 


a = .10 


AA = 200 


A^ = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


10.27 


11.18 


13.35 


10.60 


11.05 


11.55 


0.2 


36.60 


29.50 


27.03 


65.00 


52.45 


43.75 


0.4 


80.89 


71.08 


62.27 


99.30 


96.60 


93.10 


0.6 


97.60 


94.77 


90.91 


100.00 


99.95 


99.75 


0.8 


99.85 


99.47 


98.57 


100.00 


100.00 


100.00 


1.0 


99.99 


99.95 


99.91 


100.00 


100.00 


100.00 
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Table 2: Empirical power of test (in %) based on 5,000 simulations using non-Gaussian 
and non-normal £„. 



c 


a = .01 


N = 200 


N = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


2.40 


1.20 


1.85 


1.75 


1.45 


1.35 


0.2 


57.70 


46.75 


37.50 


93.75 


90.30 


82.55 


0.4 


96.90 


95.55 


91.20 


100.00 


100.00 


100.00 


0.6 


99.90 


100.00 


99.70 


100.00 


100.00 


100.00 


0.8 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 


1.0 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 




c 


a = .05 


N = 200 


N = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


8.00 


5.75 


8.15 


7.20 


5.45 


6.10 


0.2 


74.50 


64.55 


56.45 


98.55 


96.30 


92.00 


0.4 


99.40 


98.35 


96.55 


100.00 


100.00 


100.00 


0.6 


99.95 


100.00 


99.85 


100.00 


100.00 


100.00 


0.8 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 


1.0 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 




c 


a = .10 


N = 200 


N = 500 


p = 1 


p = 2 


p = 3 


p = 1 


p = 2 


p = 3 


0.0 


13.60 


12.15 


14.60 


13.60 


10.35 


11.50 


0.2 


82.30 


74.25 


65.55 


98.90 


97.70 


95.25 


0.1 


99.65 


99.10 


97.95 


100.00 


100.00 


100.00 


0.6 


99.95 


100.00 


99.90 


100.00 


100.00 


100.00 


0.8 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 


1.0 


100.00 


100.00 


100.00 


100.00 


100.00 


100.00 
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Table 3: p- values (in %) obtained by applying our testing procedure to the Tecator data set 
with p = 1, 2, and 3 principal components. 



p 


1 


2 


3 


p- value 


1.25 


13.15 


0.00 



4 Proof of Theorem 11.1 



Proof of Theorem 1.1. We have from (1.10) and (1.11) that 



Z' Z ] I Z I B 



B I + (Z^Z) Z'e 



(4.i: 



We also note that, under the null hypothesis, aij = for all i and j and therefore and £** 



of (1.8) and (1.9) reduce to 



4 =^n+ ^i(^n,Vi) 
i=p+l 



and 



1=1 



To obtain the limiting distribution of V A^A, we need to consider the vector V ( Z^Z ) Ta^s** 
We will show in Lemmas I6.2H6.7I that 







pxr 



A Opxi 



VM^ Oixp 1 / 



Op{l), 



(4.2) 



where C is an unobservable matrix of random signs, A = diag(Ai, A2, . . . , Ap), ISA = E (D^ 
and G = E (Dn.D^) , where 

D„ = vech {{{vi,X'^){vj,X'^), 1 < z < J < pf) . 



We see from (4.2) that the vector y/N |^Z^Zj Zi'^e** has the same limiting distribution as 
/ C(G-MM^)"^C 0,xp -C(G-MM^)"^CM \ 



1 ^ 

n=l 







pxr 



'rxp 

A-1 



Opxl 



i^-M^ (G - MM^) ^ Oixp 1 + (G - MM^) ^ My 
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(4.3) 



Since we are only interested in v A^A we need only consider the first r = p{p+ l)/2 elements 



of the vector in (4.3). In Lemma 6.8 we show that these are given by 



1 ^ 

= 7^ E - MM^)-' CDn - C (G - MM^)-' cm] 

"^^^ n=l 

= 7^ E (G - MM^) C (Dn - m] 

* n=l 



Then, in Lemma 6.9 we prove that 



^ ^ <* (G - MM^) C (d„ - m) A AT (^0, (G - MM^) 

n=l 



where = var {el). Finally, in Lemmas 6.10 and 6.11, we show that — = op (1). As 
a consequence of (|42|), we see that (g - MM^j - C (G - MM'^) C = op (1). Since C is a 
diagonal matrix of signs, CC = completing the proof of Theorem |1.1[ □ 



5 Proof of Theorem 11.2 



We provide only an outline of the proof since it follows the arguments used in the proof of 



Theorem 1.1 However, the arguments are simple since instead of obtaining an asymptotic 
limit distribution we only establish the weak law 



A^fG - MM^)A 



A'^'(G - MM^)A, 



(5.i: 



where A = vech ({ajj (2 — l{i = j}) , 1 < i < j < p}^) is like the vector A except without 
the random signs. 

First we note that according to Lemma 6.1 , the estimation of vi, . . . ,Vphj vi, . . . ,Vp causes 
only the introduction of the random signs Ci, . . . , Cp. As in the proof of Theorem 1.1 one can 
verify that 



A -CA 



0. 



Lemmas 6^ and 6^ hold under Hq as well as under Ha- This gives 

G - CGC = op(l) 

and 

MM^ - CMM^C = op{l), 



completing the proof of (5.1) 
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6 Technical lemmas 



Throughout the proofs in this section we will use || ■ ||i to be the 1-norm and || ■ II2 to be 
2-norm on the unit interval, square, cube, or hypercube. The null hypothesis, Hq, is assumed 
throughout this section. We will make frequent use of the following lemma, which is established 
Dauxois et al] ( [l982| ) and [B^ ( |2000| . 



m 



Lemma 6.1. // Assumptions lTll and L5 hold, then 

\\cA{t)-vm\=Op [N-^'^) 

for each 1 < i < p. 

Lemma 6.2. // Assumptions [iT7| \1.2 . and L5 hold, then there is a non-random matrix G 
such that 

(g - CGC) = op (1) , 

where G = A^~^ ^n=i -DnD^ and C, = diag (vech({QCj, 1 < i < j < p}^)) ■ 
Proof. By the Karhunen-Loeve expansion we have 



(6.1: 



Therefore an element of D„D^ is of the form y/AiAjAfcA^^j-"''^]"''^^"^^^"'''. Hence using the 
strong law of large numbers we conclude 



n=l 

where G = E (D„D^) . Thus it suffices to show that 

1 ^ 

- J2 (CDnDiC - D.D^) = op (1) . 



n=l 



(6.2) 



Expressing (6.2) elementwise, we obtain 



^ ^ (^{Xn - X, CiVi) {Xn - X, CjVj) (X„ - X, CkVk) {Xn " X, CiVi) 

- {XI V,) {XI v,){Xl Vk) {XI vA = op (1) . 



n=l 



(6.3) 



In order to prove (6.3), it is enough to show that 

N 

N 

n=l 

- {XI V,) (X:, V,) {XI Vk) {XI V,) ] = op (1) 



1 ^ 

- ((^n, CA) {XI C,V,){Xl CkVk) {XI 



(6.4) 
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and 



1 ^ 

n=l 

- (^n, C»5i)(X^, CjVj){X'^, CkVk){X^, ClVl)) = Op (1) 



(6.5) 



We only establish ( |6.4 ), since the proof of ( 6.5[ ) is essentially the same. Using Holder's 
inequality, we obtain 



X {ciVi{s)cjVj{t)ckVk{u)ciVi{w) — Vi{s)vj{t)vk{u)vi{w)) dsdtdudw 
1 ^ 



n=l 



X \\ciVi{s)cjVj(t)ckVk{u)ceVi{w) - Vi{s)vj{t)vk{u)vi{w)\\^ . 
By the law of large numbers in Hilbert spaces (cf. (Bosq, [2000 )), we have that 



1 ^ 



w 



n=l 



Op(l) 



so it remains only to show that 

\\ciVi{s)cjVj{t)ckVk{u)ceVe{w) - Vi{s)vj(t)vk{u)vi{w)\\^ = 

Using Minkowski's inequality, Fubini's Theorem, the fact that ||t'j||2 
Lemma 16. 11 we obtain 



op(l). 

= llfjlh = 1, and then 



\ciVi{s)cjVj{t)ckVk{u)ceVe{w) - Vi{s)vj{t)vk{u)ve{w)\\^ 

< \ \{CiVi{s) - Vi{s)) CjVj{t)CkVk{u)ci>Vi>{w)\\^ 

+ \ \vi{s)cjVj{t)ckVkiu) iceve{w) - Veiw))]]^ 
+ \ \Vi{s)CjVj{t) {CkVk{u) - Vk{u)) vi{w)\\^ 
+ \\Vi{s) {CjVj{t) - Vj{t))Vk{u)V(,{w)\\^ 
= \\CiVi - ViW^ + \\CjVj - VjW^ + \\CkVk - Vk\\2 + \\CiVi - ViW^ 

= Op (iV-V2) . 



Hence (6.4) is proven which also completes the proof of Lemma 6.2 



□ 



Lemma 6.3. // Assumptions \TAl and 1.5 hold, then 



1 ^ 

1J:F„DI = op(1). 



n=l 
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Proof. We see from (6.1) that an element of F„D^ can be written in the form ^^"^ , 



where F„ = ((X^, vi), {X^, V2), . . . , (X^, Vp)) ■ We observe that ^Cr ef C = °' ^° 
ing the central limit theorem, we have 

N 



i 5: F„DJ = Op (iV-'/^). 



n=l 



Repeating the arguments in the proof (6.3), one can verify that 

N 

N 



\ ( - 

C^Vj) {Xn - X, CkVk) (6.6) 

n=l ^ 

- {Xlv;){Xlv,){XlvS]= op{l) . 



Since random signs do not affect convergence to zero, the proof is complete. 
Lemma 6.4. // Assumptions \Ll\ \1.2 , and L5 hold, then 



□ 



1 ^ 

-^F„F^-A = op(l) 



n=l 



where A = diag(Ai, A2, . . . , Ap). 



Proof. By (Q, an element of F„F^ is of the form v^V^d"^^]"^- Since E^^^^^f^ = l{i = j}, 
according to the law of large numbers we have 



1 ^ 

-^F„Fl-A = op(l). 



n=l 



Thus it suffices to demonstrate that 



1 ^ 

- 5^ ((X„ - X, V.) (X„ - X, {),) - (X^, t;,) (X;:, V,)) = op (1) 



(6.7) 



n=l 



Since random signs do not affect convergence to zero, multiplying Vi by Cj and Vj by Cj will 
not affect convergence when i ^ j. If i = j, then CiCj = = 1. Therefore, it suffices to show 
that 



1 ^ 

- 5^ ((X„ - X,cA){Xn - X,c,%) - {X^^,v,){X^^,v,)) = op (1) . 



(6.8) 



n=l 



□ 



One can show ( |6.8[ ) in exactly the same way we established (6.3) in the proof of Lemma 6.2 
This completes the proof. 

Lemma 6.5. // Assumptions \Tll \1.2 , and 1.5 hold, then 



1 ^ 

]^EFn = -p(l)- 



n=l 



14 



Proof. Using (6.1), an element of F„ has the form \/Xi^i^\ so the law of large numbers 
implies that 

N 



i5:F„=op(i). 



n=l 



The proof will be completed by establishing that 



N 



N 



op (1) . 



n=l 



We express (6.9) componentwise and obtain 

N 

N 



1 ^ 

Y,{{X^^,v,)~{X^-X,v,))=op{l] 



n=l 



Since random signs do not affect convergence to zero, it suffices to show that 

N 



^ J2 UK, V^) - {Xn - X, Qt),)) = op (1) 



n=l 



We will establish (6.11) in two steps. We will show that 

N 



^J2((K,V^)-{XlcA)) = 0p{l). 



n=l 



Then, we will establish that 



1 ^ 

- J2 ((^n, CA) - {Xn - X, C,V,)) = Op [1] 



n=l 



Using the central limit theorem in Hilbert spaces with Lemma 6.1 we conclude 



N N 



n=l 



< 



n=l 

N 



n=l 



Vi - CiVi, 



1*^2 C^t^^ 1 1 2 



Op (iv-i) 



and by the same arguments we have 

N 



^ i^K, cA) - {Xn - X, CA)) 



n=l 



= \{^x- X,cA)\ 

< \\{^ix{t) - X{t)) cA{t)\\^ 

< \\fixit)-X{t)\\^ 
= op(l). 



(6.9) 



(6.10) 



(6.11) 



(6.12) 



(6.13) 



□ 
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Lemma 6.6. // Assumptions lTll \1.2 , and 1.5 hold, then 

M - M = Op (1) . 
where M = A^-^ YI1=i andm = E (D„). 
Proof. An arbitrary element of D„ is of the form 



1 ^ 

- J](X„-X,{;,)(X„-X,{),). 



n=l 



6.6 



follows 



Since this is exactly the same as the form of an arbitrary element of F„F^, Lemma 
from the proof of Lemma 6.4 Note in particular that when i ^ the sum converges to zero 
and is unaffected by signs, and when i = j, the signs cancel each other out. For this reason, 
= M, rendering it unnecessary to multiply M by in the statement of the lemma. □ 



Lemma 6.7. // Assumptions lTJ\ and L5 hold, th 



en 





/CGC 


0,xp M \ 


\ 












= op (1) 


((?) - 


Opxr 


A Opxi 




V 




Olxp 1 / 


I 





Proof. This follows immediately from Lemmas |6.2f|6.6 
We will now use Lemma 



□ 



other parameters in (1.11). 



6.7 



to separate our estimate. A, of A from the estimates of the 



Lemma 6.8. If Assumptions 1.1- L5 hold, then 



N 



M 



op a). 



n=l 



Proof. Let 





/CGC 


Orxp M \ 


c = 


Opxr 


A Opxi 






Olxp 1 / 



Using the fact that CM = M, one can verify via matrix multiplication that 

/ C(G-MM^)"^C 0,xp -C(G-MM^)"^CM \ 



C 



-1 



Opxr A Opxl 

^-M^ (G -MM^)"^ Oixp l + M^(G-MM^)"^My 
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Since 1/22^^-** jg bounded in probability, by (4.1) and Lemma 6.7 

Op(l) 



we have 



iV ( B - B I - C^iV^/^Z^e* 



We observe that C ^A^ xj^^r ^^^^ expressed as 

/ C(G-MM^)"\ 0,xp -C(G-MM^)"\m \ 



N 



n=l 







pxr 



A-1 



Opxl 



y-M^ (G - MM^) ^ Oixp 1 + (G - MM^) ^ My 



Notice that the first r = p{p + l)/2 elements of the vector in (6.15) are given by 



Br 



iV- J2 E*: (c (G - MM^) ' C Or.p -C (G - MM^) Cm) F„ 

N 

= J2 C (C (G - MM^) CD„ - C (G - MM^) CM^ 

n=l 

= ^<*C (G - MM'^)"^ C (d„ - m) . 



n=l 



Therefore 



AT 

TiVA - ^ _ MM^) C (d„ - m) = Op (1) . 



n=l 



The result is now obtained by multiplying (6.16) on the left by <^ 



Lemma 6.9. // Assumptions iLlWl . 5\ hold, then 

N 



J2 C (G - MM^) ^ C (d„, - m) ^ iV (0, (G - MM^) 



-1 



n=l 



where 



-^2 „2 
T = a 



i=p+l 



and cr^ = var e„. 

Proof. We prove this lemma in three steps. First we establish that 



N 

N-'" <* ( (^D„, - m) - (D„ - M)) = op (1) . 



n=l 
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In the second step we prove that 

N 



n=l 



i=l 



and 



N 



J2 (D„ - M) (X - /i^, c,{),) = Op (1) 



(6.18) 



(6.19) 



n=l 



Combining (6.17), (6.18), and (6.19) we obtain immediately that 



N 

iV-V2 ^ (G - MM^) (C (CD„ - m) - < (D„ - M)) = op (1) 



ri=l 



Therefore, the lemma will be established by the third step: 



N 

^ (G - MM"^)"^ < (D„ - M) A iV (o, (G - MM^) 



T\-i 



(6.20) 



n=l 



as 



We will now proceed to prove (6.17). The left side of (6.17) can be expressed elementwise 

N 

SO it is sufficient to show that 

N 

N-"'Y.<* ii^n,c^v^{K,c,v,) " {X^, V,) {X^, V,)) = Op {N^'/') (6.22) 



n=l 



and 



N 



^-1/2 J2 e** - X, cA) {Xn - X, c,v,) - (X^, cA) {X'n, c,v,)) = Op (1) . (6.23) 



n=l 



The left side of ( |6.22D is 

AT 



AT 



n=l 



n=l 



It follows from Assumptions 1.1-1.4 that both sets of random functions {e„X^(t)X^(s), 1 < 



n < N} and {X^(u)X^(t)X^(s), 1 < n < N} are independent and identically distributed with 
zero mean so by the central limit theorem in Hilbert spaces we have 



N 



N-'i'Y.^nx:,{t)x:{s) 



n=l 



Op (1) and 



N 



N-'/'J2^»^n{t)X:{s] 



n=l 



Op(l). 
(6.24) 



Next we write that 



TV 



^-1/2 j2 e*: (x;;, cA) {{x^ a,%) - (x;;, v,)) = 8, + 6, + 5, + 5,, 



n=l 



where, by (6.24), Lemma 6.1 and repeated apphcations of the Cauchy-Schwarz inequahty, we 
have 



N 



< 



< 



n=l 
N 



n=l 

N 



n=l 

Op (iV-V2) , 



\CjVj[S) -Vj[S)\\^ 



< 



N oo 

^''^'H E bk{X:„v,){X^^,cA){{X:,c,v,)-{X^^,v,)) 

n=l k=p+l 
N 



k=p+l 



N-^/'Y.^:{u)X:{t)X:{s 

n=l 

= Op {N-"') , 

N p 

N'"^ E E ^^^^n. Vk - cuVk){Xl cA) ((X;:, C,%) - {XI V,)) 



n=l fc=l 



< 



Ei^^ 

fc=i 

Op 



TV 



N~"'Y.^:{t)x:{s)x:[ 



w 



and 



1^4 



n=l 



TV p 



\\Vk{w) - CkVkiw)\\^\\CjVj{s) - Vj{s) 



n=l k=l 



< 



E E ^^^^ - ^'^■^'^) ^^^^) ((^n' ^^■%) - ^^■)) 

TV 



k=l 

Op {N"'/^) . 



n=l 



CjVj[S) -Vj[S)\\^ 



Similarly, 



N 



N-"^Y.^*:{Xlv,) {{X^^,cA) - {X:,v,)) = op(l) 



n=l 



and therefore (6.22) is proven. 
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We now establish (6.23). The left side of (6.23) is equal to 

N N 



^-1/2 J2 el* (X„ - X, cA) (/ix - X, CjV,) + AT- V2 ^ (x;;, c,%) (/XX - X, . 



n=l 

We write that 



?i=i 



N 



^-1/2 J2 e*: (X, - X, Qt),) (/ix - X, c,{), ) = 6^ + 5, + 5j + 6. 



n=l 



where, by the central limit theorem in Hilbert spaces, Lemma |6.1[ and the Cauchy-Schwarz 
inequality, we have 



N 



N ^ e„(X„ - X, CiVi) {fix - X, CjVj) 



n=l 



< |(/ix - X,CjVj)\ 

= Op {N~'/') , 



N 



n=l 



1^6 



N oo 



n=l k=p+l 

N oo 

N~'^'Y1 Yl h{Xl,Vk){Xn-X,cA) 

n=l k=p+l 
N 



< \ {fix - X,CjVj) 
= \{Hx- X.CjVj) 
= |(/ix - X,CjVj) 
= N'/'\{fix-X,c,v,) 
= N'/'X,\{fix-X,c,v,) 
= N'/^XA{fix~X,c,v,) 



N-^/^ J2 / / Kit) (X„(s) - X{s)) Us) J2 bkVkit) ds dt 

n=lJ^ k=p+l 
TV „1 „1 oo 

N~^/^Y. / / [Xr,{t)-X{t))[Xr,{s)-X{s))h{s) Yl hvkit)dsdt 

n=l-^0 -^0 k=p+l 

nl oo 
c{t,s)vi{s) Y hvk{t)dsdt 
- " k=p+i 

oo 

^i(^) Y ^kVk{t)dt 



k=p+l 



I oo 



bkVkit) {vi(t) - CiViit)) dt 



k=p+l 



<N'/^k\{fix-X,CjVj)\ 
= Op [N-y^] , 



bkVkit) 

k=p+l 



\Vi(t) - CiVi(t)\\r 
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\5, 



N p 



^ 5Z Vk - CkVk) {Xn - X, CiVi) {Hx - X, CjVj) 



n=l k=l 



< \ {fix - X,CjVj)\ 
= Op {N-'/') , 



N p 
n=l k=l 



\\Vk{t) - 44(^)11^ 



and 



1^8 



N p 



^ 5Z ^k{X - IJ,x,CkVk){Xn - X, CiVi) {Hx - X, CjVj) 



n=l k=l 



< \ {fix -X,CjVj)\ 
= Op (iV-V2) . 



^bk{X - nx,CkVk) 



k=l 



N 



n=l 



This proves (6.23), which also completes the proof of (6.21) and hence (6.17). 

We proceed to the second step, which is the proof of (6.18) and (6.19). We express (6.18) 
elementwise as 

N / p \ 

N-'" J2 i{X^„v,){X^^,v,) - = j}) [j^^kiX^^Vk - CkVk) = op (1) . (6.25) 



n=l 



,k=l 



We observe that by the central limit theorem in Hilbert spaces and Lemma 6.1 we have 

N 



N / p 

N~^I^Y.[T.^'^^^n,Vk- CkVk) 
n=l \k=l 



< 



n=l 

Op . 



\hk \ \ \vk{t) - CkVk{t)\\.^ 

2 k=l 



Similarly, 



N 



N-y^Y.^X:,''^){Xlv,) [Y^UK.Vk-CkVk) 



n=l 

p 



< 



El"' 



k=l 



N 



N-"'Y.^:{t)x:{s)x^. 



[w] 



n=l 



\\Vk{w) - CkVk{w) 



Op (iV-i/2) 



This proves (6.25) and hence (6.18). Next, we establish (6.19). We can express (6.19) elemen- 
twise as 



N 



iV-1/2 ^ ((x;^, Vk) (X;;, V,) - \kl{k = £}) (X - /ix, cA) = op (1) 



(6.26) 



n=l 



Using the previous arguments, one can easily verify (6.26), establishing (6.19). 
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We will now finish the proof of the lemma by establishing (6.20) as the third step. Using 
Assumptions |1.3| and (1.4), we see that e* has mean zero and variance given by 



oo oo 



\i=p+l j=p+l 

oo 

i=p+l 

oo 

i=p+l 

Therefore, (D„ — M) is an iid sequence with mean zero and variance (G — MM^) . 
The central limit theorem now proves (6.20), completing the proof of the lemma. □ 



Lemma 6.10. // Assumptions \T7^1.5\ are satisfied, then 

t j - [ ^ j = Op (iV-V2) . 



In particular, we have 
and 



\\bkVk{t)-hvkm\2 = 0p {N-'/^) 
\\a,,Mt)^A^)h = Op (iV-i/'), 



(6.27) 

(6.28) 
(6.29) 



where dij and hi are defined by 

A = vech ({a,,, (2 - l{z =j}),l<i<j< p}^) and B = {b,, 62, • • • , ^ ^ • 

and 6.9 imply that A = Op (A^^^/^). According to (6.14) and (6.15) we 
B - B = Op {N-^/^) , (6.30) 



Proof. Lemmas 
can prove that 



6.8 



by showing that 



or equivalently that 



We note that 



1 ^ 

lECA-^F„ = 0p(Ar-V2) 



n=l 



1 ^ 



(6.31) 



n=l 



1 ^ 

— E <*{^n - X, V,) = ^9 + 5l0 + 5ll + 5l2, 



n=l 
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where, following the arguments in the proof of Lemma 6.9, one can verify that 



1^9 



1 ^ 

— ^en{Xn-X,Vi) 



n=l 



Op {N-'/') 



10 1 



N oo 



and 



11 



^121 



N 

n=l k=p+l 
N p 



J^'^'^bk{X^,Vk- CkVk) {Xn-X,Vi) 



n=l k=l 
N p 



n=l k=l 



Op (iV-V2) ^ 
= Op (iV-V2) , 

= Op . 



This proves (6.31) and hence (6.30). 



To complete the justification of (6.27), we need to show that 

fi-j2 = 0p . 



(6.32) 



Due to (6.14) and (6.15), (6.32) will be established by proving that 

N 

N 



1 ^ 

— J2 C (-M^ (G - MM^) D„ + 1 + (G - MM^) m) = Op (N-^^^) . (6.33) 

n=l 



To prove (6.33), it is sufficient to show 



and 



n=l 



N 



i5:c=op(iv-/^). 



(6.34) 



(6.35) 



n=l 



Due to Lemma 6.9, (6.35) implies (6.34), so we prove only (6.35). We write that 



M^^n* ^ '^13 + ^14 + 5l5 + ^16, 



n=l 



where, by the central limit theorem in Hilbert spaces and Lemma 6.1, we have 



^4 



I'^isl = 



1 ^ 

ivE^« 



n=l 



Op (iV-1/2) 



N oo 



n=l k=p+l 



< 



1 ^ 



n=l 



^ bkVk{t) 
k=p+l 



Op {N-'/^) 
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151 



^ N p 

^^^bk{X^,Vk- CkVk (t) ) 



n=l k=l 



and 



1^: 



16 1 



^ N p 



n=l k=l 



Op (iv-1) 



Op {N-'/^) . 



This proves (6.35), which estabhshes (6.32) and completes the proof of (6.27). 



Using (6.27) and Lemma 6.1, we will now show (6.28) and (6.29). We conclude from (6.27) 



that 



6- - Cibi = Op and aij = Op (N-^^^) 



Now, Lemma 6.1 yields that 



Similarly, 



\\bkVk{t) - hvk(t)\\2 < \\bk{vk{t) - CkVk{t))\\2 + \\{bkCk - h)vk{t)\\2 
< \bk\\\vk(t) - CkVk{t)\\2 + \bkCk - bk\ 
= Op {N-'/^) . 



This proves (6.28) and (6.29) and completes the proof of the lemma. 



□ 



Lemma 6.11. If Assumptions ljj_ lj5_ are satisfied, then 

f2 - = Op . 

Proof. Since 



N 



N 



n=l 



it is enough to show that 



Since 



1 ^ 

j^Y.i^n-el^)=Op{N'^l^). 



(6.36) 



n=l 



1 ^ 1 ^ 1 ^ 1 ^ 



n=l 



n=l 



n=l 



n=l 



(6.36) follows from 



and 



1 ^ 



n=l 



N 



Op 



Op {N-'/') 



(6.37) 



(6.38) 



n=l 
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We decompose (6.37) as 



1 ^ 

j^^i^n- e*J e*^ = r]i + ri2 + r/3, 



n=l 



where 



Vi 



1 ^ 



n=l 
N p 



V2 



n=l 1=1 
N P P 



^3 = ^ E< EE(2 - l{z = j}) {a,,{X^„,v,){X^^,v,) - a,,(X„ - X,v,){X„ - X,v,)) 

n=l i=l j=i 

It is clear that rji = Op {N~^). We also see that 772 = 772,1 + ^72,2 + ^72,3 + ^?2,4, where 

AT p 



^2,2 



^72,3 



n=l j=l 

TV p 

n=l i=l 

N p p _ ^ 

E E E - - ^^), 

71=1 £=1 j=l 

Af p p p 

-N E EE(2 - = E {bdX'n,^^) - k{Xn - X,v,)) . 



n=l 1=1 k= 
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Applying (6.28) and the central limit theorem in Hilbert spaces we obtain that 

p 



1^2,1 1 



p 

i=l 
P 

i=l 



=1 i=l 
1 ^ 

- 5^ y; - - x{t)) v,{t) 

n=l 
1 ^ 



n=l 



1=1 



1 ^ 



n=l 



P 
i=l 



1 ^ 



n=l 



- hiVi{t) 



1=1 

+E 



AT 



i=l 



n=l 
1 ^ 



n=l 



P 

SE 

i=l 



1 ^ 



n=l 



H-E 

i=l 
P 

+E 

i=l 



1 ^ 



n=l 
AT 



biVi{t) - hiVi{t) 

\\X{t) - ^^{t)\l 



n=l 



Op (Ar-V2) _ 

In a like manner, one can verify that ?72,j = Op (X^^/^) ,i = 2,3,4. This proves that ri2 



Op {^N ^/^). In a similar fashion, one can show that 773 = Op {^N ^/^). This proves (6.37). 
Following the previous arguments, one can establish (6.38 ), completing the proof of the lemma. 

□ 
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